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Spurred by a demand to prepare more American workers for middle-class jobs in high- 
technology industries, funders are investing aggressively in large-scale partnerships 
between community college workforce educators and employers. Despite such 
investments, however, few broadly usable insights have been gleaned about effective 
partnership and workforce program implementation. That gap limits the national capacity 
for identifying the best practices of these partnership efforts—identification that is needed 
for scaling the practices up for broader use to improve workforce education programming. 


To address this problem, from 2009 to 2015, researchers 
from SRI International’s Education Division studied 
community college workforce educational implementation 
in five industries and geographic regions. 

That work has resulted in the Workforce Education 
Implementation Evaluation (WEIE), a framework for evaluating 
hard-to-measure aspects of the design, development, and 
delivery of workforce education partnerships and programs. 
Partnership strategies between employers and community 
colleges permit local regions to make adjustments in their 
workforce education system to ensure adequate supplies of 
skilled workers to meet employer demand. 

For researchers, evaluators, and workforce education 
team leaders, the WEIE framework documents and tracks 
aspects of workforce programming that make for partnership 
sustainability and appropriate instructional programs. 

To illustrate WEIE’s potential, the SRI Education Division 
study applied the framework to two contrasting cases. 

Each case represents a predominant approach to workforce 
program development in the United States: the large-scale 
partnership, an approach that involves close collaborations 
that can lead to structural changes in the college’s 
instructional delivery system; and traditional employer 
outreach, an approach that involves the college in occasional 
consultations with employers to identify workforce skills gaps. 

WEIE use clarified how regional labor market structure 
and dynamics guide decisions about how much effort 
to invest in a partnership. It highlighted the partnership 


implementation factors associated with building community 
college workforce program capacity to leverage social and 
organizational capital from employers and other institutions. 
And it revealed the instructional program implementation 
factors associated with preparing workers for the workplace 
and lifelong careers. This brief discusses those findings 
and implications for improving the delivery and evaluation 
of workforce education programming. 

The WEIE framework relies on the following research 
approaches: 

• Detailed analyses of occupational and industry 
employment databases, company data, an online 
job advertisement dataset, and a dataset consisting 
of resumes of program alumni 

• Partnership analysis drawing from interviews with more 
than 80 college leaders, industry partners, and regional 
economic development representatives in the U.S. 
Northeast, South, Midwest, and Southwest, and in diverse 
high-technology industries offering middle-class wages— 
information technology (IT), advanced manufacturing, 
biotechnology, engineering technology, and wind 
energy technology 

• Instructional data collection focusing on two high- 
technology workforce programs offered at about 10% of 
American community colleges: IT support and advanced 
manufacturing maintenance 
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Instructional analysis based on: 

• Interviews and surveys of 8 instructors in the 2 
target fields, focusing on their methods to make 
students job-ready; surveys of 38 students; 
and interviews with 25 students 

■ Observations conducted in 8 classes to 
characterize instructional methods in workforce 
education 

■ Collection of materials and student work from 
24 lessons 

■ Ratings of instructional materials and student 
work. Ratings were conducted by 12 industry 
professionals and workforce educators using 
a validated workforce rubric. 


How the WEIE Works 

The rest of this brief explains how the WEIE framework 
works by applying its tools and concepts to two contrasting 
cases, each of which reflects a common approach 
to workforce development: (1) large-scale partnership, 
and (2) employer outreach. 

In large-scale partnership one large employer or an 
allied group of employers works closely with educators to 
develop training goals; share equipment and labs; provide 
curriculum resources; and participate in training by offering 
apprenticeships and internships, and by sponsoring other 
workplace activities. 


Rationale and Core Ideas for an 
Implementation-focused Approach 

Evaluations of American postsecondary workforce 
partnerships currently emphasize outcomes and productivity. 
They focus on numbers of instructional materials produced 
and numbers of students recruited, enrolled, and job- 
certified. Measuring productivity is important, but evaluations 
focused only on such measures yield little insight about 
the implementation processes that lead to more effective 
workforce partnerships and instruction. 

The WEIE framework offers evaluation concepts and 
measurement tools to provide more data about the quality 
of workforce program implementation. It focuses on three 
key elements: 

• Labor market context - Data key to understanding 
employer partnering options 

• Partnership quality - Concepts to assist in regional 
partnership development 

• Instructional quality - Practices to deepen learning and 
build transferable skills 



Figure 2. Framework for workforce education implementation evaluation 


In employer outreach the college periodically invites industry 
representatives to meet briefly as an advisory panel to 
review workforce program goals and curriculum. This is the 
predominant approach to workforce programming. 

Labor Market Context 

Helps answer the questions: How 
do local labor market structure and 
dynamics shape workforce education 
partnering needs and approaches? How 
can program outcomes be tracked? 

The structure and dynamics of the regional labor market 
underlie what kind of workforce training is needed in a 
region, what kind of partnership opportunities are likely to 
evolve, and what vulnerabilities those partnerships will have 
to economic trends and shocks. The WEIE framework used 
analysis of traditional labor market data, i.e. government data 
on employment structures, and online job postings data, 
to characterize the labor market context of the large-scale 
partnership and employer outreach cases. The large-scale 
partnership case focused on a college program developed 
to prepare advanced manufacturing workers, and its partner 
was a major automotive manufacturer. The employer 
outreach case focused on a college program developed to 
prepare information technology support workers. 
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The structural analysis of the large-scale partnership case 
determined that the concentration of regional employment in 
program target occupations was high relative to the national 
average, with a large concentration of jobs offered in the 
automotive manufacturing industry: an industry dominated by 
the community college’s industry partner. For the employer 
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Regional Structure of Key Industries (2013) 

Case 1 - Large-Scale Partnership in Advanced Manufacturing 
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Case 2 - Employer Outreach in Information Technology 
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Regional concentration of industry sector (RC) is defined as RC = 
(industry share of regional employment / industry share of national employment). 
The key industry sectors are the five sectors that employ the most workers in the 
target occupations at the regional level, and are shaded from dark (highest 
number of workers in target occupations) to light. 

Source: SRI analysis of the Bureau of Labor Statistics (BLS) 

(2014, 2015) Quarterly Census of Employment & Wages and Occupational 
Employment Statistics Data. Data are based on Private Sector employment only. 


outreach case, jobs in program-related occupations were 
less common in the region than the national average, and 
distributed across a more diverse group of industries. These 
results illustrate how structural data can inform local college 
decisions on the number and intensity of partnerships to 
develop. 


Analysis of labor market dynamics was needed to 
complement structural analysis. Are industries growing 
or contracting, how many workers are they hiring, in what 
occupations and with what skills and credentials? To assess 
dynamics, the WEIE framework recommends review of 
trends in traditional employment data and in online job 


advertisements, including the frequency of job postings and 
salary levels. This analysis draws on a job advertisement 
database developed by Rothwell (2014). For the large-scale 
partnership advanced manufacturing case, the total number 
of workers employed in program-related occupations had 
been growing, but most of the key industries that employed 
these workers had actually contracted in recent years. In 
addition, the key partner had reduced staff and did not seem 
to be placing online job want-ads. In the employer outreach 
IT case, the occupations students were being prepared 
for were experiencing high growth nationally, but regional 
employment had been more volatile and growth across 
key industries was mixed. However, we saw a high level of 
online job posting activity in the IT occupations, indicating 
that regional demand might have been stronger than the 
structural analysis suggested. Understanding labor market 
dynamics helps colleges customize their outreach and 
programming to help graduates build skills that are regionally 
in-demand by employers. 

Outcomes for workforce program participants can be 
drawn from resume databases that characterize the career 
trajectories and skills of graduates. By pairing this information 
with job-posting data we can also characterize demand for 
graduate skills in the regional labor market. While rigorous 
implementation of such an approach would require tracking 
and collecting resume data from graduates, this is in fact 
significantly less burdensome than analyses that require 
the use of sensitive student data and access to protected 
Unemployment Insurance (Ul) databases: factors which 
can be prohibitively time-consuming, complex and costly 
for routine evaluation. An exploratory application of the 
approach, based on convenience samples that should not 
be considered representative of all program graduates, 
indicated that for both cases about one third of sampled 
program alumni's current or most recent employment was 
in a target occupation. In addition, our analysis showed 
moderate regional demand for the skill-set of the alumni 
in the large-scale partnership sample, and strong regional 
demand for the skill-set of the alumni in the employer 
outreach partnership sample. Such outcome data can 
be used by the college to describe the marketability of 
program graduates. 
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Large-scale partnership can increase the number of organizations in program 
support roles and decrease the number of organizations in less supportive roles. 


Different Roles of 

Regional Institutions* 

Case 1: Large-scale Partnership 
with Key Partner 

Advanced Manufacturing 

Case 2: Multiple Employer 

Outreach Approach 

Information Technology 

Resource Providers 

One industry partner is primary resource provider 

Shares a wide range of resources: space, 
equipment, curriculum, faculty, knowledge 

Multiple employers 

Each shares some knowledge 
and curriculum resources 

Competitors 

No direct competition 

2-year and 4-year colleges 

Short-term certification programs 

Complementary organizations 

State and regional industry sector associations; 
Pre-collegiate engineering program; K-12 schools; 
4-year colleges; Regional economic development 
agencies 

Regional IT training resource and technical 
assistance center 

Beneficiaries 

One industry partner mostly 

Multiple employers and industry sectors 

Opponents/problem makers 

Academic and other workforce programs within 
the college 

Academic and other workforce programs within the 
college 

Other colleges and trade schools 

Affected bystanders 

Advanced manufacturing employers in the region 

Pre-collegiate STEM programs; K-12 schools; 
Regional economic development agencies; 

Business associations 


*Framework adapted from Arney, Eddy, & Campbell (2010) and Bloom & Dees (2008) 


Partnership Quality 

Helps answer the questions: What roles 
can different regional institutions play to 
enhance workforce educators’ capacity 
to respond to labor market needs and 
economic cycles? What partnership 
strategies should be monitored? 

Partnerships need to be responsive to political forces, 
administrative structures, and varying cultural values, both 
locally and regionally. The WEIE builds on ideas such as 
partnership capital (Amey, Eddy, & Campbell, 2010) and 
social ecosystems (Bloom & Dees, 2008). These concepts 
can help guide partnerships to bring about systemic change. 
To do so, the framework focuses on 1. identifying the roles 
different regional institutions should play in relation to each 
other and 2 . developing practices of monitoring partnership 
strategies that lead to success. 


Identifying Roles the Partners Play 
in a Regional Ecosystem 

Building partnership both reduces competition and can help 
secure social and organizational resources for the workforce 
education program. The table on this page shows how 
partnership roles and interactions with other organizations 
affect the quality of partnership. 

Monitoring the Four Key Partnership Strategies 

A workforce partnership uses four strategies to set the goals 
of the partnership and improve its sustainability. Monitoring 
them can strengthen a partnership: 

• Coalitions building expands two kinds of workforce 
program support: social (e.g., lobbying, engaging potential 
competitors) and organizational (e.g., equipment, space) 

• Communications processes establish the vision, values, 
and priorities for the partnership's approach to systemic 
change in workforce preparation 

• Credibility development involves having both short¬ 
term and long-term plans to demonstrate the workforce 
program's effectiveness to external audiences 
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• Contingencies preparations ensure the workforce 
program has both the social support and organizational 
resources to weather changes in labor markets and 
funding opportunities offered by government, industry, 
education, and other players 


Partnership 

Strategies* 

Case 1: 
Large-Scale 
Partnership with 

Key Partner 

Case 2: 
Multiple 
Employer 
Outreach 
Approach 

Coalitions 

</+ 

- 

Communications 

•/+ 

- 

Credibility 

•/+ 

✓- 

Contingencies 

s- 



*Strategies adapted from Bloom & Dees, 2008 


Budgets and time for workforce instructional design and 
delivery are tight. Accordingly, implementation tools that lead 
to more consistent quality of instructional approach have high 
value. The WEIE focuses on four instructional implementation 
quality indicators drawn from the literature of technician 
education, adult learning, and training (Boucouvalas & 
Lawrence, 2010; Burke & Hutchins, 2007; Felder, Woods, 
Stice, & Rugarcia, 2000; Finch & Crunkilton, 1999; Herman, 
Bramucci, Fiala, & Litman, 2000; Keiser, Lawrenz, & 

Appleton, 2004; Kerka, 1997; Lualhati, 2007; Torraco, 2008). 
These four factors are Industry relevance , academic rigor , 
applied practice , and proof of performance. The WEIE 
provides tools for checking both the intended curriculum 
and the enacted curriculum (Porter, 2002, 2004). Using 
these tools helps workforce educators focus instructional 
programming appropriately along the continuum between 
“knowledge-level preparedness” and “transfer-level 
preparedness.” 


The table above shows how these processes look from the 
perspectives of the large-scale partnership case and the 
employer outreach case. The large-scale partnership exceled 
at coalitions, communications, and credibility, but could 
be seen as more mixed on contingencies. Its close ties to 
one industry make it vulnerable to labor market change but 
its strong profile on the other three strategies enhances its 
readiness to take advantage of new funding opportunities. 
The employer outreach case, with its personal links to a 
few different industries and its stance of serving as many 
industry sectors as possible, was somewhat effective in 
building credibility but in a mixed position with respect to 
contingencies. For example, its broad preparation approach 
puts it in a good position to weather labor market change, 
but possibly not as strong a position to exploit new funding 
opportunities. 

Instructional Quality 

Helps answer the question: How can 
the instructional design and delivery 
choices of workforce educators improve 
preparation for transferable knowledge 
and skills? 


■ 



■ 

w 

Knowledge-Level 

Transfer-Level 



Preparedness 

Preparedness 

A 

1 


■ I A 

fl 


Industry Relevance: Exam-based 
Academic Rigor: Basic terms/procedures 
Applied Practice: Structured familiarization 
Proof of Performance: Checklist/example practice 
Workplace preparedness quality continuum 


Workplace task-based 
Basic & strategic thinking 
Case-based reasoning 
Industry performance rubric 


Research Methods 

Interviews with workforce program leaders and instructors 
provided data on the intended curriculum. Expert panel 
ratings of a sample of instructional materials (24 lessons) 
and observer ratings of instructional methods (8 lessons) 
provided enacted curriculum data. The 12 experts were 
industry professionals and workforce educators who were 
trained in rating the instructional materials using a 0-4 
scale of the Technicial Education Curriculum Assessment 
(TECA) (Keiser, Lawrenz, & Appleton, 2004). Analysis 
entailed creating subscales for different workforce quality 
attributes and then examining differences between beginning 
and advanced courses in the two programs. Observers 
characterized the quality of applied practice opportunities. 
They tallied the class times devoted to instructor-led activities 
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(e.g., lectures) and student-led activities (e.g., hands-on 
labs); and they rated the depth of learning (Webb, 2002) 
of these two types of activities, as well as the degree of 
problem solving complexity (Jonassen, 2000) these activities 
entailed. Analysis compared the differences on these 
features between beginning and advanced courses in the 
two programs. 

Balance Between Knowledge- and Transfer- 
Preparedness in Instructional Materials 

Intended curriculum data indicated: Representatives of the 
large-scale partnership emphasized transfer-preparedness, 
as reflected in the factory space the employer partner 
donated for training; their desire to integrate coursework with 
the student internships they offered; and their suggestions for 
class activities that employed workplace technologies and 
simulated work tasks and time pressure. Representatives 
of the employer outreach program emphasized knowledge- 
preparedness more, with focus on preparing students for 
industry certification exams; at the same time, they also 


suggested integrated class activities that relied on workplace 
technologies and on situations that simulated work tasks and 
time pressure. 

As described below, enacted curriculum analysis produced 
a more nuanced portrait of the two cases. 

Instructional Materials Quality 

Expert ratings of the instructional materials and student 
work for both programs show how the four quality 
indicators-industry relevance, applied practice, 
academic rigor, and proof of performance-contributed to 
an overall understanding of the enacted curriculum in terms 
of the workplace preparedness quality continuum from 
knowledge-level focus to transfer-level focus. 


Instructional Materials Ratings on Four Quality Indicators and Associated 
Placement on Workplace Preparedness Quality Continuum 


Program 

Knowledge-focused 
implementation placement 

Mixed implementation placement 

Transfer-focused 
implementation placement 

Large-scale 

partnership 

Proof of performance 

Academic rigor 

Industry relevance* 

Applied practice 

Employer outreach 

Proof of performance 

Industry relevance 

Applied practice 

Academic rigor* 

Specific program notes on reasons for materials ratings 


Large-scale 

partnership 


Lab assessments focused too much 
on task completion and didn’t rate 
facets of technical acumen. More 
situational judgment tasks needed. 


Adequate academic rigor achieved 
through focus on scientific principles 
(e.g., simple machines and electrical 
circuits), and mathematics (e.g., to 
calculate resistance and set ranges 
for robotic programming). 


Internships, factory classroom, and 
updated equipment strongly supported 
industry relevance. 

Applied practice at all course levels 
offered a mix of guidance with varied 
opportunities to troubleshoot. 


Employer outreach Lab assessments focused too 

much on task completion and didn’t 
rate facets of technical acumen. 

Few situational judgment tasks 
were included. Online quizzes and 
lengthy textbook readings were 
overemphasized. 


Adequate industry relevance 
achieved through preparation for 
industry certification exam. 

Applied practice was sufficiently 
varied in the advanced course, but 
only adequate in beginning courses, 
which mostly used “cookbook labs.” 


Strong academic rigor achieved 
through focus on the scientific 
principles (e.g., electrical circuits) and 
mathematics (e.g., translating binary 
to hexadecimal notations; assigning 
IP addresses). Lab reports required 
explanations based in principles. 


‘Denotes significant difference between programs 
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Instructional Methods Quality 

Observers rated the quality of instructional methods 
that support applied practice, with focus on whether 
implementation practices led to students showing classroom 
behaviors associated with deep learning and complex 
problem solving. These measures help characterize the 
quality of hands-on labs. 

Instructional Methods Ratings on Two Factors of Applied Practice Quality and Associated 
Placement on Workspace Preparedness Quality Continuum 


Program 

Knowledge-focused 

Mixed implementation placement 

Transfer-focused 


implementation placement 


implementation placement 




High problem complexity in advanced 

Large-scale 

Basic problem complexity in 

Adequate deep learning opportunities 

courses 

partnership 

beginning courses 

in beginning courses 

Many deep learning opportunities in 

advanced courses 


Basic problem complexity in 

Medium problem complexity in 


Employer outreach 

beginning courses 

advanced courses 


Few deep learning opportunities in 

Adequate deep learning opportunities 



beginning courses 

in advanced courses 



Implications 


The WEIE framework can deepen understanding of 
implementation practices, resulting in improved effectiveness 
for workforce-employer education partnerships: 

• Labor market analysis of both structure and dynamics 
identifies gaps in a region between the skills employers 
require and the jobs that workers are prepared to do. It 
also indicates the market growth potential for workers 
trained in the skills of target fields. Community college 
leaders can use this information to plan employer outreach 
strategy and instructional programming. 

• Labor market analysis also offers a low-burden way 
of characterizing partnership outcomes through the 
compilation and analysis of alumni resumes. Resumes 
can be collected directly from alumni or via searchable 
online resume databases to determine if alumni have 
obtained jobs in their field of study, and what their skill-sets 
are. When combined with job-advertisement data, this 
analysis is also useful for describing the transferable skills 
developed by the program that are in demand regionally 
and nationally. 

• Whether workforce educators opt to embark on a 
large-scale partnership or not, the WEIE’s partnership 


implementation approach provides for identifying how 
various other players in the region relate to a college 
workforce program and how to develop an outreach 
strategy that enhances both social and organizational 
support for the program. It also indicates the primary 
partnership strategies to monitor over time to improve 
the partnership network, strengthen its message, build 
credibility by showing short-term and long-term results, 
and stay ready for economic and political contingencies. 

• Instructional implementation analysis singles out four 
criteria to monitor in order to improve the transfer- 
level focus of a workforce education program: industry 
relevance, academic rigor, applied practice, and proof of 
performance. It also provides tools for rating instructional 
materials in regard to these four criteria and for rating 
instructional methods focused on applied practice (e.g., 
hands-on labs) in terms of the two factors of problem 
complexity and depth of learning. These concepts and 
tools may be used to refine instructional programs as 
part of an industry advisory panel meeting, a working 
collaboration with employers around lesson design, faculty 
meetings to select industry-created materials, or periodic 
college program reviews for accreditation purposes. 
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Tools and other information available at: 

http://www.sri.com/work/projects/ 

community-college-partnerships-instructional-impacts 


This material is based upon work supported by the National 


Science Foundation under Grant No. DUE 0903331. Any 
opinions, findings, and conclusions or recommendations 


expressed in this material are those of the authors) and do 


Suggested citation: Yarnall, L., Tennant, E., & Stites, R. 
(2015). A framework for evaluating implementation of 
workforce education partnerships and programs. Research 
Brief. Menlo Park, CA: SRI International. 


not necessarily reflect the views of the National Science 
Foundation. 
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